Generating an Image of the Surroundings of an Articulated Vehicle

ABSTRACT

Systems and methods for generating an image of the surroundings of an articulated vehicle are provided. According to an aspect of the invention, a processor determines a relative position between a first vehicle of an articulated vehicle and a second vehicle of the articulated vehicle; receives a first image from a first camera arranged on the first vehicle and a second image from a second camera arranged on the second vehicle; and combines the first image and the second image based on the relative position between the first vehicle and the second vehicle to generate a combined image of surroundings of the articulated vehicle.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.14/247,710, filed Apr. 8, 2014, the entire disclosure of which is hereinexpressly incorporated by reference.

BACKGROUND OF THE INVENTION

The present invention relates to systems and methods for generating animage of the surroundings of an articulated vehicle, in which differentparts of the vehicle are capable of moving with respect to each other.An example of an articulated vehicle is a tractor-trailer in which atractor pulls a trailer. The tractor and the trailer may travel indifferent directions at times, such as when the tractor begins to turn acorner.

Related art systems use multiple cameras arranged on a single-bodyvehicle to generate a birds-eye view image of the surroundings of thevehicle. For example, U.S. Pat. No. 7,139,412 discloses a system inwhich a plurality of cameras are mounted on a vehicle such that there issome overlap between the fields of view of adjacent cameras. Separateimages acquired by the plurality of cameras are combined to generate abirds-eye view image of the surroundings of the vehicle.

In these related art systems, the locations of the cameras are fixedwith respect to each other, because they are mounted at fixed positionson a single-body vehicle. Existing algorithms for the related artsystems cannot combine the images if the cameras change their relativepositions, such as if one camera is mounted on the tractor and anothercamera is mounted on the trailer, and the tractor and trailer aretraveling in different directions. However, in order to provide completecoverage of the surroundings of the articulated vehicle, it would beadvantageous to arrange cameras on both the tractor and the trailer.

SUMMARY OF THE INVENTION

Exemplary embodiments of the invention provide systems and methods forgenerating an image of the surroundings of an articulated vehicle.According to a method of the invention, a processor determines arelative position between a first vehicle of an articulated vehicle anda second vehicle of the articulated vehicle; receives a first image froma first camera arranged on the first vehicle and a second image from asecond camera arranged on the second vehicle; and combines the firstimage and the second image based on the relative position between thefirst vehicle and the second vehicle to generate a combined image ofsurroundings of the articulated vehicle.

The relative position between the first vehicle and the second vehiclemay be determined based on an angle between the first vehicle and thesecond vehicle, and the angle may be measured by an angular sensorarranged on the articulated vehicle.

Alternatively, the relative position between the first vehicle and thesecond vehicle may be determined based on a first motion of the firstvehicle and a second motion of the second vehicle, the first motion maybe measured by at least one first sensor arranged on the first vehicle,and the second motion may be measured by at least one second sensorarranged on the second vehicle. The first motion and the second motionmay be used to determine an angle between the first vehicle and thesecond vehicle. Each of the first motion and the second motion mayinclude a vehicle speed, a wheel speed, a yaw rate, and/or anacceleration.

As another alternative, the relative position between the first vehicleand the second vehicle may be determined by detecting at least onefeature of the second vehicle that appears in at least two images fromthe first camera, or the relative position between the first vehicle andthe second vehicle may be determined by detecting at least one featureof the first vehicle that appears in at least two images from the secondcamera. A first one of the images from the first camera may be acquiredwhen an angle between the first vehicle and the second vehicle is zero,and a second one of the images from the first camera may be acquiredwhen the angle between the first vehicle and the second vehicle isnon-zero.

The relative position between the first vehicle and the second vehiclemay be determined continuously. The first image and the second image maybe combined by rotating the first image and the second image withrespect to each other, based on an angle between the first vehicle andthe second vehicle. The first image and the second image may betransformed into ground plane images before rotating the first image andthe second image.

The processor may also superimpose information from an active sensingsystem on the combined image, and components of the active sensingsystem may be arranged on the first vehicle and/or the second vehicle.The components may include ultrasonic sensors and/or radar sensors. Theinformation may be color-coded to indicate a type and a relevance ofobjects detected by the active sensing system. The processor may displaythe combined image and a forward-view image, wherein the forward-viewimage is acquired by a forward-facing camera arranged on the firstvehicle. The combined image may be a birds-eye view image of thesurroundings of the articulated vehicle.

According to another aspect of the invention, a system for generating animage of the surroundings of an articulated vehicle is provided. Thesystem includes a first camera arranged on a first vehicle of anarticulated vehicle; a second camera arranged on a second vehicle of thearticulated vehicle; a memory; and a processor coupled to the memory.The processor includes position determining logic that determines arelative position between the first vehicle and the second vehicle;image receiving logic that receives a first image from the first cameraand a second image from the second camera; and image combination logicthat combines the first image and the second image based on the relativeposition between the first vehicle and the second vehicle to generate acombined image of surroundings of the articulated vehicle.

According to yet another aspect of the invention, a non-transitorycomputer-readable medium including computer instructions executable by aprocessor to cause the processor to perform the methods discussed aboveis provided.

Other objects, advantages, and novel features of the present inventionwill become apparent from the following detailed description of theinvention when considered in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a block diagram of an exemplary embodiment of a system forgenerating an image of the surroundings of an articulated vehicle;

FIG. 2 shows a flowchart of an exemplary embodiment of a method forgenerating an image of the surroundings of an articulated vehicle;

FIG. 3 shows an example of an articulated vehicle having camerasarranged on the tractor and the trailer;

FIGS. 4A and 4B show an angle between the tractor and the trailer as thetractor begins to make a turn;

FIG. 5 shows another example of an articulated vehicle having camerasarranged on the tractor and the trailer;

FIG. 6 shows an example of combining two images based on the anglebetween the tractor and the trailer;

FIG. 7 shows an example of a ground plane into which the images may betransformed;

FIG. 8 shows an example of detecting a three-dimensional object withinthe images; and

FIG. 9 shows an embodiment in which an active sensing system is used tosuperimpose information about objects in the surroundings of thearticulated vehicle on the birds-eye view image.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

FIG. 1 shows a block diagram of an exemplary embodiment of a system forgenerating an image of the surroundings of an articulated vehicle, whichincludes at least two vehicles that may move with respect to each other.As shown in FIG. 1, the system includes a memory 10 and a processor 20coupled to the memory 10. The processor 20 includes logic 30-50, whichwill be described in more detail below in connection with FIG. 2. Theprocessor 20 can be any type of processor, such as a microprocessor, afield programmable gate array (FPGA), and/or an application specificintegrated circuit (ASIC). When the processor 20 is a microprocessor,logic 30-50 can be processor-executable code that is loaded from thememory 10.

FIG. 2 shows a flowchart of an exemplary embodiment of a method forgenerating an image of the surroundings of the articulated vehicle. Forexample, as shown in FIG. 3, the articulated vehicle may include a firstvehicle 200 that is a tractor and a second vehicle 210 that is atrailer. A plurality of cameras 220, 230, and 240 may be arranged on thefirst vehicle 200, and a plurality of cameras 250, 260, and 270 may bearranged on the second vehicle 210. For example, the cameras may facedownward to obtain a birds-eye view image. Alternatively, the camerasmay be ultra-wide angle cameras with fisheye lenses, such as a Sunex®lens with an Omnivision® high dynamic range sensing chip. Anyappropriate number and arrangement of cameras may be used, and theconfiguration shown in FIG. 3 is only one non-limiting example.Advantageously the cameras may be arranged to provide complete 360°coverage of the surroundings of the articulated vehicle. In addition,more than one trailer may be attached to form a road train.

As discussed above, because the first vehicle 200 and the second vehicle210 are capable of moving with respect to each other, the positions ofthe cameras 220, 230, and 240 arranged on the first vehicle 200 are notfixed with respect to the positions of the cameras 250, 260, and 270arranged on the second vehicle 210. On the contrary, as shown in FIG.4A, as the first vehicle 200 starts to turn, an angle is formed betweenthe first vehicle 200 and the second vehicle 210, and the relativepositions of the cameras 220, 230, and 240 arranged on the first vehicle200 change with respect to the positions of the cameras 250, 260, and270 arranged on the second vehicle 210.

Accordingly, as shown in FIG. 2, the position determining logic 30determines the relative position between the first vehicle 200 and thesecond vehicle 210 of the articulated vehicle at step 100. The relativeposition may be determined by any appropriate method, and may bedetermined continuously or at predetermined intervals. As shown in FIG.4B, the angle α between the first vehicle 200 and the second vehicle 210may be determined. For example, an angular sensor arranged on thearticulated vehicle may be used to directly measure the angle a betweenthe first vehicle 200 and the second vehicle 210. Alternatively, varioussensors arranged on the first vehicle 200 and/or the second vehicle 210may be used to measure the motion of the first vehicle 200 and/or thesecond vehicle 210, such as individual wheel speed sensors, yaw ratesensors, and/or accelerometers. Further, the vehicle speed may bedetermined from the wheel speed sensors, from the engine, or from anyother appropriate source. Measurements from these sensors may be inputto a model of the articulated vehicle, such as a bicycle model, in orderto determine the angle a between the first vehicle 200 and the secondvehicle 210.

As yet another alternative, the angle a between the first vehicle 200and the second vehicle 210 may be determined by using the cameras 220,230, 240, 250, 260, and/or 270 to detect common features in imagesrecorded by the cameras. For example, as shown in FIG. 4A, the camera240 may protrude from the first vehicle 200 and have a downward-lookingfisheye lens. Accordingly, the field of view of the camera 240 includeseverything below the height of the camera 240 and extends out to thehorizon, including parts of the second vehicle 210, such as theleft-front trailer edge, the wheels, and/or markings on the side. Thesefeatures can be used to determine the angle a between the first vehicle200 and the second vehicle 210, as discussed in further detail below.

For example, the camera 240 may collect images while the angle a iszero, such as when the articulated vehicle is traveling straight down ahighway. These images indicate where the features appear on average. Forexample, the average of the center of the wheels may be determined,and/or the average image column in which the left-front trailer edgeappears may be determined. As an alternative, a geometrical model may beused to determine where the features appear while the angle a is zero.When the first vehicle 200 turns, the angle α can be determined based ona comparison of the features in the current image with the featureswhile the angle α is zero. For example, the angle α may be determined bycomparing the position and/or the shape of the wheels. This analysis mayalso account for characteristics of the camera 240, such as the focallength of the lens, and the known geometry of the articulated vehicle.

Returning to FIG. 2, the image receiving logic 40 receives imagesacquired by the cameras arranged on the first vehicle 200 and the secondvehicle 210, respectively at step 110. The cameras may be configured toacquire images simultaneously and at the same frame rate. The fields ofview of the cameras may have some overlap. FIG. 5 shows a simplifiedexample in which camera 300 has a field of view 330, camera 310 has afield of view 340, and camera 320 has a field of view 350. In thisexample, the fields of view 330 and 340 overlap near a central axis ofthe articulated vehicle, and the field of view 350 overlaps with thefield of view 330 and/or the field of view 340 behind the second vehicle210.

The image combination logic 50 combines the received images based on therelative position between the first vehicle 200 and the second vehicle210 at step 120. Referring to FIG. 4A, the images may be combined bystitching together images acquired by the cameras 220, 230, and 240arranged on the first vehicle 200 to generate a first image; stitchingtogether images acquired by the cameras 250, 260, and 270 arranged onthe second vehicle 210 to generate a second image; and stitchingtogether the first image and the second image after aligning the firstimage and the second image based on the angle α between the firstvehicle 200 and the second vehicle 210.

FIG. 6 shows a first example in which the first image 500 and the secondimage 510 are combined by defining a junction between the first image500 and the second image 510 based on the angle α between the firstvehicle 200 and the second vehicle 210. In this example, portions of thefirst image 500 and/or the second image 510 are removed in the region520 where the first image 500 and the second image 510 overlap to form acombined image. The removed portions are determined by the geometry ofthe system, including the angle α.

As a second example, the first image and the second image may becombined by translating and rotating the images with respect to eachother based on the angle α between the first vehicle 200 and the secondvehicle 210. The first image and the second image are converted intoground plane images by applying the homography matrix H of therespective camera to the image according to H*(px py 1), where px and pyare the locations of the pixels in the image plane. The homographymatrix H may be defined during a calibration phase according to knownmethods. FIG. 7 shows an example of a ground plane, in which the groundplane projection of each pixel P within an image is positioned at adistance R from the kingpin K and forms an angle p with respect to thelongitudinal axis 530 of the second vehicle 210. A rotation is thenapplied to the ground plane coordinates, such that the rotated laterallocation is given by R*sin(ρ+α) and the rotated longitudinal location ofthe pixel P is given by R*cos(ρ+α). Once this process has been completedfor both images, the images are combined as described above withreference to FIG. 6. It is possible to use only a portion of the firstimage 500, only a portion of the second image 510, or portions of boththe first image 500 and the second image 510 for the overlap region 520.

Because the angle α changes as the first vehicle 200 turns, the imagesmay be acquired continuously or periodically, and may be combined basedon the changing angle a. Accordingly, the combined image of thesurroundings of the articulated vehicle may be updated continuously orperiodically. The examples discussed above result in a birds-eye viewimage of the surroundings of the articulated vehicle. However, theexamples may be modified to produce a surround view of the surroundingsof the articulated vehicle.

The methods described above may also be used to alert the driver of thearticulated vehicle that there is an object in the surroundings of thearticulated vehicle. FIG. 8 shows a first image 600 and a second image610 that have been converted into ground plane images. As shown in FIG.8, the first image 600 and the second image 610 can be translated androtated with respect to each other in order to align the lane markingson the parking lot, because the lane markings are within the groundplane. However, it is not possible to align the person 620 standing inthe first image 600 and the second image 610, because the person 620projects above the ground plane in three dimensions. This informationcan be used to warn the driver to avoid the person 620 when maneuveringthe articulated vehicle.

FIG. 9 shows another embodiment in which an active sensing system isused to superimpose information about objects in the surroundings of thearticulated vehicle on the birds-eye view image. For example, activesensing components, such as ultrasonic sensors and/or radar sensors, maybe arranged on the first vehicle 200 and/or the second vehicle 210.These active sensing components may detect objects 410, such aspedestrians, other vehicles, and/or obstacles. Representations of theseobjects 410 may then be superimposed on the birds-eye view image of thesurroundings of the articulated vehicle. The representations of theobjects 410 may be color-coded in order to indicate the type andrelevance of the objects 410. For example, the relevance may indicatewhether the articulated vehicle is likely to collide with the object410.

The birds-eye view image may include the current state of thearticulated vehicle, including the current angle a between first vehicle200 and the second vehicle 210. Further, the birds-eye view image may bedisplayed with a forward-view image of an area 280 in front of the firstvehicle 200, as shown in FIGS. 3 and 5. In addition, path predictions420 and 430 for the second vehicle 210 and the first vehicle 200,respectively, may be included, as shown in FIG. 9. Maneuvering guidance440 may also be included, as shown in FIG. 9.

Although the embodiments described above obtain images from camerasarranged on the first vehicle 200 and the second vehicle 210 of anarticulated vehicle, a similar method could be applied to camerasarranged on different parts of a single vehicle that move with respectto each other. For example, a tractor may include a cabin mounted onsprings, such that the cabin can move with respect to a rigid base ofthe tractor. The relative position between the cabin and the rigid basemay be determined, and images from a camera mounted on the cabin may becombined with images mounted on the rigid base to provide a combinedimage of the surroundings of the tractor, according to the generalprinciples discussed above.

The methods discussed above are executed by a computer processor that isprogrammed to perform the methods so that the processor executes theprogramming to perform the methods. Such a processor is needed to handlethe large volumes of data and to perform the complex andcomputationally-intensive analysis of the methods discussed above. Inaddition, the processor is required to perform the methods in acommercially viable timeframe. Accordingly, it is necessary to quicklyprocess large and complex data sets.

According to another exemplary embodiment of the invention, there isprovided a non-transitory computer-readable medium encoded with acomputer program for generating an image of the surroundings of anarticulated vehicle. The term “computer-readable medium” as used hereinrefers to any medium that participates in providing instructions forexecution. Common forms of computer-readable media include, for example,a floppy disk, a flexible disk, a hard disk, magnetic tape, any othermagnetic medium, a CD-ROM, any other optical medium, punch cards, papertape, any other physical medium with patterns of holes, a RAM, a PROM,an EPROM, a FLASH-EPROM, any other memory chip or cartridge, and anyother non-transitory medium from which a computer can read.

The foregoing disclosure has been set forth merely to illustrate theinvention and is not intended to be limiting. Since modifications of thedisclosed embodiments incorporating the spirit and substance of theinvention may occur to persons skilled in the art, the invention shouldbe construed to include everything within the scope of the appendedclaims and equivalents thereof.

What is claimed is:
 1. A method comprising: determining, by a processor,an angle between a first vehicle of an articulated vehicle and a secondvehicle of the articulated vehicle as the first and second vehiclesrotate laterally relative to each other around a point at which thefirst and second vehicles are connected to each other, based on arelative position between a first camera arranged on the first vehicleand a second camera arranged on the second vehicle, the first and secondcameras being located on a same side of the articulated vehicle;receiving, by the processor, a first image from the first cameraarranged on the first vehicle and a second image from the second cameraarranged on the second vehicle, the first and second images beingobtained from the same side of the articulated vehicle; and combining,by the processor, the first image and the second image based on therelative position between the first camera and the second camera togenerate a combined image of surroundings of the articulated vehicle onthe same side thereof, wherein the first image and the second image arecombined by rotating the first image and the second image with respectto each other, based on the angle between the first vehicle and thesecond vehicle.
 2. The method according to claim 1, wherein the angle ismeasured by an angular sensor arranged on the articulated vehicle. 3.The method according to claim 1, wherein the angle between the firstvehicle and the second vehicle is determined based on a first motion ofthe first vehicle and a second motion of the second vehicle, the firstmotion is measured by at least one first sensor arranged on the firstvehicle, and the second motion is measured by at least one second sensorarranged on the second vehicle.
 4. The method according to claim 3,wherein each of the first motion and the second motion comprises atleast one of a vehicle speed, a wheel speed, a yaw rate, and anacceleration.
 5. The method according to claim 1, wherein the anglebetween the first vehicle and the second vehicle is determined bydetecting at least one feature of the second vehicle that appears in atleast two images from the first camera.
 6. The method according to claim5, wherein a first one of the images from the first camera is acquiredwhen the angle between the first vehicle and the second vehicle is zero,and a second one of the images from the first camera is acquired whenthe angle between the first vehicle and the second vehicle is non-zerodue to the first and second vehicles rotating laterally relative to eachother around the point at which the first and second vehicles areconnected to each other.
 7. The method according to claim 1, wherein theangle between the first vehicle and the second vehicle is determined bydetecting at least one feature of the first vehicle that appears in atleast two images from the second camera.
 8. The method according toclaim 7, wherein a first one of the images from the first camera isacquired when the angle between the first vehicle and the second vehicleis zero, and a second one of the images from the first camera isacquired when the angle between the first vehicle and the second vehicleis non-zero due to the first and second vehicles rotating laterallyrelative to each other around the point at which the first and secondvehicles are connected to each other.
 9. The method according to claim1, wherein the relative position between the first vehicle and thesecond vehicle is determined continuously.
 10. The method according toclaim 1, wherein the first image and the second image are transformedinto ground plane images before rotating the first image and the secondimage.
 11. The method according to claim 1, further comprisingsuperimposing information from an active sensing system on the combinedimage, wherein components of the active sensing system are arranged onat least one of the first vehicle and the second vehicle.
 12. The methodaccording to claim 11, wherein the components comprise at least one ofultrasonic sensors and radar sensors.
 13. The method according to claim11, wherein the information is color-coded to indicate a type and arelevance of objects detected by the active sensing system.
 14. Themethod according to claim 1, further comprising displaying the combinedimage and a forward-view image, wherein the forward-view image isacquired by a forward-facing camera arranged on the first vehicle. 15.The method according to claim 1, wherein the combined image is abirds-eye view image of the surroundings of the articulated vehicle. 16.A system comprising: a first camera arranged on a first vehicle of anarticulated vehicle; a second camera arranged on a second vehicle of thearticulated vehicle, wherein the first and second cameras are arrangedon a same side of the articulated vehicle; a memory; and a processorcoupled to the memory, the processor comprising: position determininglogic that determines an angle between the first vehicle and the secondvehicle as the first and second vehicles rotate laterally relative toeach other around a point at which the first and second vehicles areconnected to each other, based on a relative position between a firstcamera arranged on the first vehicle and a second camera arranged on thesecond vehicle, the first and second cameras being located on a sameside of the articulated vehicle; image receiving logic that receives afirst image from the first camera and a second image from the secondcamera, the first and second images being obtained from the same side ofthe articulated vehicle; and image combination logic that combines thefirst image and the second image based on the relative position betweenthe first camera and the second camera to generate a combined image ofsurroundings of the articulated vehicle on the same side thereof,wherein the first image and the second image are combined by rotatingthe first image and the second image with respect to each other, basedon the angle between the first vehicle and the second vehicle.
 17. Anon-transitory computer-readable medium comprising computer instructionsexecutable by a processor to cause the processor to perform a methodcomprising: determining an angle between a first vehicle of anarticulated vehicle and a second vehicle of the articulated vehicle asthe first and second vehicles rotate laterally relative to each otheraround a point at which the first and second vehicles are connected toeach other, based on a relative position between a first camera arrangedon the first vehicle and a second camera arranged on the second vehicle,the first and second cameras being located on a same side of thearticulated vehicle; receiving images from a first camera arranged onthe first vehicle and a second camera arranged on the second vehicle,the first and second images being obtained from the same side of thearticulated vehicle; and combining the first image and the second imagebased on the relative position between the first camera and the secondcamera to generate a combined image of surroundings of the articulatedvehicle on the same side thereof, wherein the first image and the secondimage are combined by rotating the first image and the second image withrespect to each other, based on the angle between the first vehicle andthe second vehicle.